Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffNemo_bis Date: Nov 3, 2013 3:23am
Forum: web Subject: Re: Allow IA to index, prevent search engines

On a standard website, the easiest approach would be to put those you want to be archived in a different directory, use robots.txt instead of meta tags and whitelist the Wayback machine for that directory (though I don't know if Allow: is accepted by the machine), User-agent: ia_archiver according to https://archive.org/about/exclude.php

Terms of Use (31 Dec 2014)